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Abstract 

In the first paper of this two part communication, we solved in a unified framework a 
variety of two terminal source coding problems with noncooperative encoders, thereby 
consolidating works of Shannon, Slepian-Wolf, Wyner, Ahlswede-Korner, Wyner-Ziv, 
Berger et al. and Berger-Yeung. To achieve such unification we made use of a funda- 
mental principle that dissociates bulk of the analysis from the distortion criterion at 
hand (if any) and extends the typicality arguments of Shannon and Wyner-Ziv. In this 
second paper, we generalize the fundamental principle for any number of sources and on 
its basis exhaustively solve all multiterminal source coding problems with noncoopera- 
tive encoders and one decoder. The distortion criteria, when applicable, are required to 
apply to single letters and be bounded. Our analysis includes cases where side informa- 
tion is, respectively, partially available, completely available and altogether unavailable 
at the decoder. As seen in our first paper, the achievable regions permit infinite order 
information-theoretic descriptions. We also show that the entropy-constrained multi- 
terminal estimation problem can be solved as a special case of our theory. 
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1 Introduction 



In the first paper of this two part communication [1], we solved in a unified framework a 
variety of two terminal source coding problems with noncooperative encoders, consolidating 
works of Shannon [2, 3], Slepian-Wolf [4], Wyner [5], Ahlswede-Korner [6], Wyner-Ziv [7], 
Berger et al. [8] and Berger-Yeung [9] . In particular, we derived a fundamental source coding 
principle extending the typicality arguments of Shannon [3] and Wyner-Ziv [7], and, using 
this principle, showed inner bound properties on the achievable regions. We also showed the 
outer bound properties using interposed lossless coding (as seen in [7]) and Fano's inequality 
[10]. In this second paper, we extend our framework to multiterminal source coding with 
noncooperative encoders. Specifically, we exhaustively enumerate twelve problems in three 
categories where encoded sources are decoded 1) losslessly, and 2) under distortion criteria, 
respectively, and 3) a subset of the encoded sources are decoded losslessly whereas the rest are 
decoded under distortion criteria. In each category, one of the following four subcases arises. 
At the decoder either 1) side information is unavailable, or 2) side information is available 
at a certain rate (partially), or 3) side information is available completely, or 4) part of 
the side information is available partially whereas part is available completely. We shall see 
that the eleven remaining problems are special cases of the problem where only a subset 
of the sources are losslessly decoded whereas the rest are decoded under distortion criteria 
with part of side information available partially and part available completely. This problem 
sans side information has Berger-Yeung problem [9] as its two terminal specialization. We 
solve the general multiterminal version of the above problem using our usual methodology. 
Specifically, we give an infinite order description of the achievable region. The inner bound 
is shown using a multiterminal extension of our two terminal fundamental principle that 
dissociates bulk of the analysis from distortion criteria and extends typicality arguments of 
Shannon [3] and Wyner-Ziv [7]. The outer bound is shown using interposed multiterminal 
lossless coding and Fano's inequality extending our two terminal argument [1]. Finally, we 
shall demonstrate that the scope of our theory extends beyond the traditional source coding. 
In particular, we shall solve the entropy-constrained estimation problem in a multiterminal 
setting as a special case of our theory. We organize our analysis as follows: We pose the 
twelve multiterminal source coding problems in Sec. 2 and present their solutions in Sec. 3. 
In Sec. 4, we state and prove the general multiterminal version of the fundamental principle 
of source coding. The proof of our general source coding theorem is given in Sec. 5. We 
apply our theory to multiterminal entropy-constrained estimation in Sec. 6. Finally, Sec. 7 
concludes the paper. 
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2 Multiterminal Problems 



We begin with an exhaustive enumeration of multiterminal source coding problems where 
individual encoders do not cooperate. In the process, we bring out the similarity, the dis- 
similarity and the interdependency among such problems. We also identify the problems, 
which have already been solved completely, which have been solved in special cases and for 
which certain bounds have been found. Subsequently, we shall solve the unsolved problems 
in their most general setting. For the sake of convenience, we pose distributed source coding 
problems in a phased manner: Basic source coding (without side information) in Sec. 2.2, 
source coding with partial side information in Sec. 2.3 and source coding with complete side 
information in Sec. 2.4. First we need some notation and the concept of strong typicality. 



2.1 Notation 

Throughout this paper we denote random variables by uppercase letters such as X, Y, Z, 
and their alphabets by corresponding script letters X, y, Z. All alphabets are finite unless 
otherwise stated. By H(X) and I(X;Y), denote entropy of X and mutual information 
between X and Y, respectively. Further, by Ik, denote the set {1,2, ...,K}. We adopt the 
convention I = {}. Also, denote j + I = {j + m : m G /} . Clearly, M + Ik = Im+k \ Im- 
Denote the k-th element of a sequence by x(k), the corresponding sequence by {x(k)} and 
the collection of all elements indexed by h\ through h 2 by x{h\\ h^). Also write x n = x(l; n) 
and x n (k) = x(n(k — 1) + l;nk). Denote vector (collection) (Xi, X 2 , Xm) of random 
variables by X, and the corresponding alphabet by X — X\ x X 2 x ... x Xm- Moreover, 
denote by /, the vector of mappings f m : X m — > Z m , m G Im- Here the fact that / has 
component functions with distinct domains and ranges will sometimes be indicated by the 
symbol '/ : X >— » Z\ For any / C I M , denote by Xj the vector of {X m : m G /} and 
by Xj = Ylmei ^ m denote the corresponding alphabet. Also, denote by ] l the vector of 
mappings {f m } m ei- Further, denote Rj = J2mei (note the contrast with R T , the vector 
of {R m : m G /}). In addition, denote the closure of set A by A. Finally, define the 
e-strongly (e > 0) typical set of X ~ p(x) by [10] 

T} n \X) = |x n G X n : ^-N(x\x n ) - p(x) < ^ for all x G # J , (2.1) 

where N(x\x n ) denotes the number of occurrences of x in the sequence x n . In this paper, 
we consider only strong typicality which will henceforth be mentioned simply as typicality. 
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Consequently, we have, for sufficiently large n (due to strong law of large numbers), 

Pr{X n i T € (n \X)} < e, (2.2) 

where {X(k)} are drawn i.i.d. ~ p(x). Also if x n G % in) (X), then we call x n a typical 
sequence. In an analogous manner, the jointly typical set of a collection of random variables 

X = (Xi, X 2 , X M ) is defined by (2.1) with X, x and X replaced by X, x and X = 

X I x X 2 x ... x X M) respectively. 



2.2 Basic Source Coding 

Consider vector of M random variables X = (Xi,X 2 , ...,1m) ~ p(%), components of which 
are separately encoded and jointly decoded. Specifically, draw {X(k)} i.i.d. ~p(x), encode 
X using M encoder mappings 

f m :X^^Z m , meI M (2.3) 
(i.e., / : X n h-> Z) for some alphabet Z and decode using decoder mapping 

g : Z - X n . (2.4) 

We call 

T = g(fQT)) (2.5) 

the estimate or reconstruction of X n . Further, a rate M-vector R is said to be achievable if 
for any e > 0, there exists (for n sufficiently large) mapping pair (/,<?) such that 

- log \Z m \ < R m + e, me I M (2.6) 

n 

^ n 

and appropriate error or distortion criteria based on (X , X ) are met also within an accuracy 
of e. 

Depending on such criteria, we enumerate three sub-problems (and assign each a tag, 
e.g., 'L') in the following. 

I. Lossless Coding ('L'): X is losslessly decoded (in the sense of Shannon). Specifically, 
a rate vector R is said to be achievable if for any e > 0, (2.6) holds alongside 

PrpT ^ X"} < e. (2.7) 

Denote by A L the set of achievable R. This problem has been solved by Shannon [2] 
for M = 1 and by Slepian and Wolf [4] for general M. 
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2. Coding under Distortion Criteria ('D'): X is decoded under L bounded distortion 
criteria di : X — > [0, dj max ], I € h- The achievable set *4. D is defined by the set of pairs 
(R, D) (D being an L-vector) such that, for any e > 0, (2.6) holds alongside 



-Ed ln (T, T) < Di + e, I eh (2.8) 

Th 



where 



di n (x n ,x n ) = ^ di(x k , x k ) 
k=i 

2n 

(of course, di n : X — > [0, ndi max \). The special case, where M = 1 and L — 1, was 
solved by Shannon [3]. Also, the case, where M = 2 and L = 1, was solved in our first 
paper [1] of this series. 

3. Lossless Coding in a Subset ('T'): A subset Xj, J C J M , of sources X, is losslessly 
decoded and the complementary subset Xjc is decoded under L bounded distortion 

2 

criteria d\ : X Jc — > [0,d; max ], / G Jl- The achievable set A T is defined by the set of 
(R, D) pairs such that, for any e > 0, (2.6) holds alongside 

PrpTj^Tj)} < e (2.9) 

1 „ 

-Ed ln (X JC ,X JC ) < Dt + e, I eh. (2.10) 

n 

The special case, where M = 2, J = {1} and L — 1, was solved by Berger and Yeung 
[9]. 

Note that problem 'T' is the same as problem 'L' for J = I M and as problem 'D' for J = {}. 
Next we generalize the basic source coding problem to incorporate side information. 



2.3 Partial Side Information 

First consider encoding of X using partial side information. Specifically, suppose (X, W) ~ 
p(x,w) (W being a K- vector) and draw {(X(k), W(k))} i.i.d. ~p(x,w). Now encode (X,W) 
using M + K encoder mappings 

fm '■ <%m —* Zmi TU G Im (2 11) 

fM+m ■ -> Z Af+m , m e h 

(i.e., / : ^Y™ x W™ >— > Z) and decode using decoder mapping g : Z — > ^Y™ as in (2.4). In 
other words, now estimate X™ by 

f = ff (/(r,r)). (2.i2) 
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Note that only partial knowledge of side information W is available at the decoder (W n , 
however, is not estimated). Further, a rate (M + K)-vector R is said to be achievable if for 
any e > 0, there exists (for n sufficiently large) mapping pair (/, g) such that 



and appropriate error or distortion criteria based on (X™, X ) are met also within an accuracy 
of e. Specifically, we modify the three basic source coding problems enumerated in Sec. 2.2 
(each problem tag is now appended with 'P') as follows. 

We shall refer (2.7)-(2.10) below; in each case, assume reconstruction X = g(f(X n , W™)) 
as given in (2.12). 

1. Lossless Coding ('LP'): The achievable set A LP is defined by the set of R such that, 
for any e > 0, (2.13) holds alongside (2.7). The special case, where M — 1 and K — 1, 
was solved by Wyner [5] and Ahlswede-Korner [6]. 

2. Coding under Distortion Criteria ('DP'): The achievable set A OP is defined by 
the set of (R,D) pairs such that, for any e > 0, (2.13) holds alongside (2.8). In the 
special case, where M = 1, K = 1 and L — 1, an inner bound on A OP was found by 
Berger et. al [8] and a complete solution was derived in our first paper [1] of this series. 

3. Lossless Coding in Subset ('TP'): The achievable set A TP is defined by the set of 
(R,D) pairs such that, for any e > 0, (2.13) holds alongside (2.9) and (2.10). 

Of course, 'P' is removed from any of the abovementioned tags if W is deterministic. Next 
we consider the case where additional side information is completely available at the decoder. 

2.4 Complete Side Information 

Suppose (X,W,S) ~ p(x,w,s) (S being a scalar) and draw {(X(k), W(k), S(k))} i.i.d. 
~ p(x,w,s). Now encode (X,W) using M + K encoder mappings as in (2.11), i.e., / : 
X n x W™ >— > Z; however, decode using decoder mapping 



- log \Z m \ < R m + e, me I m +k 



n 



(2.13) 



n 



g:ZxS n ^X n . 



(2.14) 



In other words, estimate X by 



X =g(f(T,W n ),S n )- 



(2.15) 
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Accordingly, we modify the four partial side information problems enumerated in Sec. 2.3 
to also incorporate S (each tag is further appended with £ C) as follows. 



We refer (2.7)-(2.10) below; in each case, assume reconstruction X = g(/(A 7 "',W"), S n ) 
as given in (2.15). 

1. Lossless Coding ('LPC'): The achievable set A LPC is defined by the set of R such 
that, for any e > 0, (2.13) holds alongside (2.7). 

2. Coding under Distortion Criteria ('DPC'): The achievable set A DPC is defined 
by the set of (R, D) pairs such that, for any e > 0, (2.13) holds alongside (2.8). 

3. Lossless Coding in Subset ('TPC'): The achievable set v4. T pc is defined by the set 
of (R, D) pairs such that, for any e > 0, (2.13) holds alongside (2.9) and (2.10). 

Of course, £ C is removed from any of the abovementioned tags if S is deterministic. On 
the other hand, as seen in Sec. 2.3, 'P' is removed from any of the above tags, if W is 
deterministic. Correspondingly, problems 'LC, 'DC and 'TC arise, where the only side 
information S is completely available at the decoder. Note that Slepian-Wolf theorem solves 
Problem 'LC completely [4]. Also, the special case of problem 'DC (lossy coding with 
complete side information), where M = 1 and L — 1, was solved by Wyner and Ziv [7]. 

2.5 Summary 

So far we have identified twelve source coding problems in Sees. 2.2, 2.3 and 2.4. In partic- 
ular, we divided these problems into three categories: Lossless coding ('L*'), coding under 
distortion criterion ('D*') and lossless coding in subset ('T*'). Here V is one of blank, 'P', 
'C and 'PC. For quick reference, salient features of all the twelve problems are summarized 
in Table 2.1. In this paper, we solve all the abovementioned problems in their most general 
setting, save problems 'L' and 'LC, which are already completely solved. 

Recall that problem £ T*' reduces to problem £ L*' if J = I M and to problem £ D*' if J = {} 
which we indicate by the diagram 




(2.16) 



£ L*' 



£ D*'. 
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Category 


lag 


Estimate 

- 77, 

) 


Achievability 
Conditions 


Solution 
Status 




£ L' 


<?(7PO) 


(2.6), (2.7) 


Solved: General M. 


Lossless 


£ LC 


g(f(X n ),S n ) 


(2.6), (2.7) 


Solved: General M. 


Coding 


'LP' 


affix", w")) 


(2.13), (2.7) 


Solved: M = 1, K = 1. 




£ LPC 


(?(7(x n ,Tr"),5") 


(2.13), (2.7) 


Unsolved. 




'D' 


<?(7PO) 


(2.6), (2.8) 


Solved: M = 1,2, L = 1. 


Lossy 




yV( A ) 


(2.6), (2.8) 


bolvecl. ikz = 1, .L = 1. 


Coding 


'DP' 




(2.13), (2.8) 


Solved: 
M = 1, K = 1, L = 1. 




'DPC 


g(f(X n ,W n ),S n ) 

J \ J \ 7 / 7 / 


(2.13), (2.8) 


Unsolved. 


Lossless 




g(7(x n )) 


(2.6), (2.9), (2.10) 


Solved: 
M = 2, J= {1}, L = l. 


Coding 


£ TC 


g(7(x n ),s n ) 


(2.6), (2.9), (2.10) 


Unsolved. 


in Subset 


'TP' 


gQix^W 1 )) 


(2.13), (2.9), (2.10) 


Unsolved. 




'TPC 


gifiX^W 2 )^-) 


(2.13), (2.9), (2.10) 


Unsolved. 



Table 2.1: Summary and solution status of source coding problems. 



Also recall that the problem dependency due to side information can be depicted by 

£ ?PC 

deterministic S ^/ deterministic 



£ ?P' 

deterministic TU 



£ ?C 



(2.17) 



deterministic S 



'?' 



where £ ?' is one of £ L', £ D' and T'. In view of the dependencies (2.16) and (2.17), it is enough 
to solve problem £ TPC alone. The solution can then be specialized in order to solve other 
problems. 
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3 Unified Coding Theorem 



We begin by giving a generic description of the solutions of all the twelve source coding prob- 
lems. In particular, let A be the generic notation for the achievable rate or rate-distortion 
regions defined in Sec. 2. Note that, since A is defined by appropriate e-achievability condi- 
tions (e > 0), A is closed. We state this in a unified coding theorem: 

Theorem 3.1 A = A* . 

Here A* = U^Li -Ki- We need to specify A* n for each problem which we take up next. In 
the process, we shall see that each A* n is closed. However, that does not necessarily imply A* 
is closed. Hence the closure appears in Theorem 3.1. In the following, we shall first specify 
A^ TPC corresponding to problem 'TPC, which we then specialize to the rest of the problems. 



3.1 Lossless Coding in Subset 

Problems 'TPC and 'TP': First consider problem 'TPC. A rate-distortion pair (R G 
R M+K ,D G R L ) is said to belong to A* nTPC if there exist product of (M + K) alphabets ~Z 
(with the restriction Z m = X^, m G J), conditional distributions g m (z m |a:^), m G J c , and 
rj(zM+j\wj ), j G Ik, and mapping ip : Z x S n — > A 7 ^ such that 

hpr^ZjlZj^^S 11 ) < R T , ICI M \{} (3.1) 
-I(W r ;-, Z M+I \Z M+lKXI ,S n ) < R M+I , IQI K \{} (3.2) 

71/ 

-d ln (X n Jc ^(Z,S n )) < D h I eh (3.3) 
n 

where Zj = Xj, 

(T, r, S n ,Z lM+K \j) ~ Pn (x n ,w n , s n ) n q m (z m \x n J J] rj (z M+j \w?) (3.4) 

and^^WV 1 ) = YF k=1 p(x{k),w(k), s{k)). Denote X' = (X,W) such that % M = X and 
-^m+Z/c = Then, by (3.4), U — > — > forms Markov chain for any / C 7 M+i ^ and 
any subcollection [/ of (X' ,Z,S n ) excluding (x'j ,Zi). Further, we show in Appendix A 
that, splitting each I — I' U I" C 7 M \ {} such that I' C J and J" C J c , we can equivalently 
write (3.1) as 

-//(X^lXj^,, ^/ M+K \(ju/") ; »S' n ) + ~-^(^"' ^"I^J' Zi m +k\(Jui"), S n ) < Rp + Ri». (3.5) 
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Note that the total number of conditions given by (3.1) (or, equivalently, (3.5)) and (3.2) 
is (2 M + 2 K -2). Further, consider Problem 'TP' and define A* nTP = A* nTPC such that S is 
deterministic in (3.1)-(3.4) (or, in (3.5) instead of (3.1)), i.e., occurrences of S n (and s n ) are 
simply removed. 

Problems 'TC and 'T': First consider problem TC and define *4.* TC = *4.* TPC such that 
W is deterministic in (3.1)-(3.4). Note that the left hand side in (3.2) is now zero, i.e., it is 
enough to consider only R = (R 1: R 2: ...,Rm) € M M . Also, by (3.4), Z m+Ik is independent 
of (X ,S n ,Z lM ), hence occurrences of components of Z m+Ik can be removed from (3.1), 
(3.3) and (3.4). Writing afresh, a rate-distortion pair (R e R M ,D e R L ) belongs to A* nTC if 
there exist product of M alphabets Z (now playing the role of the abovementioned Z Im ), 
conditional distributions q m {. z m\ x ^)i m e Im, an d mapping if; : Z x S n — * ~X n J c such that 

-I(Xi\Zi\Zi^S n ) < R z , IQI M \{} (3.6) 
n 

-d ln (X Jc ,ij(Z,S n )) < D h I eh (3.7) 
n 

where Zj = Xj and (X™, S n , Zjc) ~ p n (x", s n ) U m& jc Qm(z m \x^). Referring to (3.5), (3.6) 
can equivalently be written as 

H(X ji I X j\p , Zjc\j/i, S n ) H I(Xj„] Zjrr\Xj, Zjc\Jir, S™) < -R/' + (3-8) 

n N n 

where, as earlier, we split I — I' U I" C 7 M \ {} such that I' Q J and 7" C J c . Further, 
consider Problem 'T' and define ^4* T = A* nTC such that 5 is deterministic in (3.6) and (3.7) 
(or, in (3.8) instead of (3.6)), i.e., occurrences of S n (and s n ) are simply removed. Note that, 
in the special case, where M = 2, J = {1}, L — 1 and S n is deterministic, (3.8) and (3.7) 
are the same as conditions (6.28)-(6.31) of [1], which define A* n for Berger-Yeung problem 
[9]. 



3.2 Lossless Coding 

Problems 'LPC and 'LP': Now consider problem 'LPC, which is problem 'TPC with 
J = I M - In this case, (3.5) takes the form 

—H(Xj \X IC , Z m+Ik , S n ) < R[ 

because now /' = I C I M \ {}, I" = {} and I m +k \(JU I") = M + I K . Of course, (3.3) 
does not arise because distortion criteria d^s are no longer defined. Hence, writing (3.5), 
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(3.2) and (3.4) afresh, a rate vector R G R M+K belongs to *4.* LPC if there exist product of K 
alphabets Z (now playing the role of abovementioned Z m+Ik ) and conditional distributions 

rj(zj\wj'), j G Ir, such that 

-H(Tj\X n j^Z,S n ) < R T , ICI M \{} (3.9) 

n 

1 „ _ _ 

-/(^jZjIZjc,^) < i? M+/ , /C/ K \{} (3.10) 

n 

where 

(T l ,W n ,S n ,Z)^ Pn (x n ,w-,s n ) J] r^K) (3.11) 

and p n (x n , w n , s n ) = Ylk=i Pfi(k), Note that, for any / C J M and /' C i^, 

(X^, 5") — > W^/ — ► -Zr is Markov chain. Also note that the total number of conditions 
given by (3.9) and (3.10) is (2 M + 2 K — 2). Further, consider Problem 'LP' and define 
^4* LP = *4.* LPC such that S is deterministic in (3.9)-(3.11), i.e., occurrences of S n (and 
s n ) are simply removed. Note that, in the special case, where M — 1, K — 1 and S 1 ™ is 
deterministic, (3.9) and (3.10) are the same as conditions (6.14) and (6.15) of [1], which 
define A* n for the so-called "side information problem" [5, 6]. 

Problems 'LC and 'L': Consider problem £ LC and define A* nljC = *4.* LPC such that W is 
deterministic in (3.9)— (3.11). Note that the left hand side in (3.10) is zero, i.e., it is enough 
to consider only R = (R ± , R 2 , R M ) G R M . Also, by (3.11), Z in independent of (F, S n ), 
hence Z can be removed from (3.9), i.e., we have 

-HQT^T^ S n ) = H(X I \X I o, S) <Rj, IC I M \ {}. (3.12) 

Hence, observe that A* nljC = A[ LC , i.e., Al c = Al c = Al LC . In view of this, Theorem 
3.1 is a version of Slepian-Wolf theorem [10]. Further, consider Problem 'L' and define 
-4* L = A* lL = A* lLC with deterministic S in (3.12). In this case, Theorem 3.1 is the usual 
statement of Slepian-Wolf theorem. 

3.3 Coding under Distortion Criteria 

Problems 'DPC and 'DP': Next consider problem 'DPC, which is problem 'TPC with 
J = {}. Rewriting (3.1)-(3.4) for this special case, a rate-distortion pair (R G R M+K ,D G 
R L ) belongs to A* mPC if there exist product of (M+K) alphabets Z, conditional distributions 
Qm(zm\Xm)i m e Im, and rj(z M +j\Wj), j G Ik, and mapping ip : Z x S n — > X a such that 

-/(X; ; Z 7 |Z WU ,^) < Rj, ICI M \{} (3.13) 
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-KW^Zm+^Zm+j^j^S™) < R M+ i, IQIk\{} (3.14) 

77 

-dmQT^iZ.S^) < D h leh (3.15) 

77 

where 

(T 1 ,W n ,S n ,Z)^ Pn (x n ,w n ,s n ) J] q m (z m \x n m ) J] r^+>?) (3.16) 

and p„(x", tU™, s n ) = nfc=iP(^(^)5^(^)> s (^))- Again note that the total number of condi- 
tions given by (3.13) and (3.14) is (2 M + 2 K -2). Further, consider Problem 'DP' and define 
i* DP = *4.* DPC such that S is deterministic in (3.13)— (3.16), i.e., occurrences of S n (and s n ) 
are simply removed. 

Problems 'DC and 'D': Consider problem 'DC and define A* mc = -4^ DPC such that W is 
deterministic in (3.13)— (3. 16) . Note that the left hand side in (3.14) is zero, i.e., it is enough 
to consider only R = (i? 1; R 2 , Rm) £ M. M . Also, by (3.16), Zm+i k in independent of 
(A 7 ™, S n , Zj M ), hence components of Zm+i k can be removed from (3.13), (3.15) and (3.16). 
Writing afresh, A rate-distortion pair (R G R M , D e R L ) belongs to A* nuc if there exist 
product of M alphabets Z (now playing the role of the abovementioned Zj M ), conditional 
distributions g m (^ m |x™ ), m G Im, and mapping if) : Z x S n — > X n such that 

1 — n — — 

-IQTjiZjIZjc^ 11 ) < R f , ICI M (3.17) 

-^di n (X n , if)(Z, S n )) < D h I eh (3.18) 

where 

(A",W",^,Z)~p n (^,W m , S n ) J] q m {z m \x n m ). (3.19) 

Further, consider Problem 'D' and define A* m = *4.* DC such that S is deterministic in (3.17)— 
(3.19), i.e., occurrences of S n (and s n ) are simply removed. 

It is enough to prove Theorem 3.1 for problem 'TPC, which, as we have just seen, 
specializes to Theorem 3.1 for each of the rest of the problems at hand. We present the 
proof in Sec. 5, which requires a fundamental principle of multiterminal source coding that 
generalizes our earlier results given in Theorem 3.1 and Lemma 3.5 of [1]. We first state and 
prove this generalized principle (Theorem 4.1) in Sec. 4. 
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4 Fundamental Principle 



4.1 Statement 

Theorem 4.1 Let (Y,~Z,V) = (Y 1: Y 2 , Y M >, Z u Z 2 , V) fte a collection o/2M' + l 
random variables ^ p'(jj^v)Yl^ =1 q' m (z m \y m ) and let {(Y(k),V(k))} bei.i.d. copies of (Y,V). 
Then for any rate M' -vector R such that 

I (Yf, Zj|Zjc, V) <i?; (4.1) 

„/ 

/or a// / C J M / \ {} and /or any e' — > ; iaere exists a sequence of mapping pairs (f : y >— > 
U,g :UxV ^ Z ) for some sequence U of M 1 -fold product of alphabets (and some n' — > oo^) 
sitc/i t/iat 

log |Z4| < i^ + e", ie/ M ' (4.2) 

n' 

Pr{£} < e" (4.3) 

tt>aere 

^ = {(r',f ri ',\/"')^^ n ' ) (> 7 ,^^)}, 

Z = g(f(Y ),V n ) andt" -> 0. 

Here note that : 3^f' — ► Wj, i G Jm', i-e., encoders do not cooperate. Also note that 
U — > Y j — > Z [ forms Markov chain for any I C I M i and any subcollection U of (Y, Z, V) 
excluding (Yi,Z{). Further, due to strong law of large numbers, in the above (Y™ , V n ') 

can be replaced, without loss of generality, by any (Y ,V n ) such that Pr{(Y , V n ) ^ 
T} (Y,V)} < e", where e" — > as e' — > 0. Such substitutions are standard and will 
sometimes be carried out without explicit mention. Also, note that as e' — > 0, n' — > oo 
through values n' > ng(e') for appropriate n' (-). 

Theorem 4.1 roughly states the following. Using a sequence of codes (/, g) (of sufficiently 
large length n'), one can achieve any rate M'- vector R satisfying the 2 M ' — 1 inequalities 

given by (4.1) such that the estimate Z = g(/(F" ),V n ') of Z n , based on the encoding 
fiY™ ) and side information V n ' , is jointly typical with T 7 " with high probability. Further, 
Theorem 4.1 includes Slepian-Wolf 's direct theorem as a special case. To see this, set Z = Y 
and let V be deterministic, so that I (Fj; Zi\Zic, V) = H(Yj\Yjc) in (4.1). Hence, for any 

rate vector R such H(Y I \Y I c) < R' T , I C 7 M , \ {}, (4.2) holds and Pr{F ^ Y } is 
arbitrarily small (due to (4.3)). Next we turn to the proof of Theorem 4.1. 
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4.2 Necessary Ingredients 

Before proceeding any further, let us point out that we derived in Lemma 3.5 of our earlier 
work [1] a special case of Theorem 4.1 where M' = 2 and (Z 2 , V) is deterministic. In 
fact, this special case was demonstrated to encapsulate the essence of Wyner-Ziv's typicality 
argument. In other words, proving Theorem 4.1 amounts to generalizing an earlier result 
which we reproduce below for ease of reference. 

Lemma 4.2 [1, Lemma 3.5] Let (Yi,Y 2 ,Zi) ~ p'{yi , 2/2)91 (<zi \ui) and draw {(Yi(k),Y 2 (k))} 
i.i.d. ~ p'(yi,y2)- Then for any rate R[ such that 

KX^Z^) <R[ (4.4) 

and any e' — > 0, there exists a sequence of mapping pairs 

for some sequence of alphabets U\ (and some n! — > 00) such that 

- t log I Wi I < R' 1 + e" (4.5) 

TV 

^{{J x n \Y 2 n \zi)iTf\Y^Z x )} < e" (4.6) 

where if = gifiiYf'), Y 2 n ') and e" -> 0. 

Note that Y2 — > Yi — > ^i is required to form Markov chain for Lemma 4.2 to apply. 
Further, we shall require another crucial result. In particular, denote by B* the convex 
rate region defined by the conditions given in (4.1). We need to exhaustively identify the 
corner points of £>*, which are rates R such that M' of the 2 M ' — 1 constraints in (4.1) 
are active. For this purpose, we shall make the mild assumption that the random variables 
(Y, Z, V) are all dependent. Under this assumption, any Markov property of any subset of 
(Y, Z, V) does not hold if such property does not follow directly from the generic definition 
{Y,Z,V) ~ p\y,v)Y[^ =1 q> m {u m \y m ). 

Lemma 4.3 Let II be the set of permutations of (1, 2, M'). Further, denote by ir(i) the 
permuted position of i e lw under permutation ir e II. Also denote 7r(J) = '■ i G /}. 

Then B* has M'\ corner points R'*^ indexed by tt G II such that 

K(i) } = Z n{ i)\Z Ah _ lh V), ieI M >. (4.7) 
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The proof of Lemma 4.3 is somewhat involved and is relegated to Appendix B. Corre- 
sponding to the identity permutation 7Ti, from (4.7), we have 

R'* (ni) = (I(Y 1 ;Z 1 \V), I(y 2 -Z 2 \Z u V), I{Y mi -Z m/ \Z ImI _ v V)) (4.8) 

as we shall also see in Lemma B.10. Further, by Lemma 4.3 and referring to (4.1), any 
R 1 G B* can be written as 

J2KK M <K i^lM> (4.9) 
Tren 

for some {A^-j^n such that each A^ > and XLen ^ = 1- For our analysis, we shall also 
require the Markov lemma. In the following we give a version that rewords Lemma 14.8.1 of 
[10] and appears in its present form in Lemma 3.7 of [1]. 

Lemma 4.4 [10, Lemma 14-8.1] [1, Lemma 3.7] Let (Yi,Y 2 , Z\) ~ p'(yi, 2/2)^1 (^1 an d 
the sequence of triplets {(Yi(k),Y 2 (k), Zi(k))} be such that, for any e' — > (and appropriate 
n' — > 00), 

Pr{(Y 1 n ',Y 2 n ')^Tj n '\Y 1 ,Y 2 )} < e[ 

for some e[ — > 0. Then 

Pt{(Y?',Y?',Z?') i T) n '\Y 1 ,Y 2 ,Z 1 )} < e' 2 
for some e 2 such that e 2 — > as e' — > 0. 

Now we are ready to complete the proof of Theorem 4.1. 

4.3 Proof of Theorem 4.1 

Denote by B the set of rate vectors R' such that for any e' — > there exists a sequence of 
mapping pairs (/, p) satisfying (4.2) and (4.3). Clearly, Theorem 4.1 states B* C £>. Next 
we claim £> is convex. To see this, note that if each of two rate vectors r'^ and R'^ belongs 
to B, then, by appropriate time sharing, we can ensure that any convex combination of such 
vectors also belongs to B. 



15 



Lemma 4.5 Any R' , such that 

R'*^ = j(y i; ZilZj^V) <R[, ie I M > (4.10) 

(7Ti being identity permutation) , belongs to B. 

Proof. Let us begin by noting that condition (4.10) for % — 1 is same as condition (4.4) 
with (Yi,V,Zi) playing the role of (Y~i,Y2,Zi) (V — > Yi — > Zi, of course, forms Markov 
chain). Hence, by Lemma 4.2, for any e' — > 0, there exists a sequence of mapping pairs 
(/i : yf ^U l ,g 1 :U l x V n ' -> 2?') such that 

^log|Wi| < i^ + ei (4.11) 

Pr{5x} < ei (4.12) 

where 

£ 1 = {(Y?,Z n i,V"')tT< n '\Y 1 ,Z 1 ,V)}, 

Zl =g 1 (f 1 (Y 1 n '),V n ') and e[ - 0. 

In fact, for any e' — > 0, next we show that there exists a sequence of mapping pairs 
(fi ■ yf -> Z4 ^ : x x V n ') -> Zf') for each i G J M ' such that 

i log m < R\ + e\ (4.13) 
Pr{£} < e\ (4.14) 

where 

£ = {(KK vn ') i ry\Y h ,z h ,v)}, 

Z s = gi(fi(Y t n ), (Z 7 ._ , )) and — > 0. We have already seen that the above result holds 
for 2 = 1. We shall show this for general % G Im 1 by induction. Specifically, we assume that 
the result holds for % — 1 (in place of %) for some % G {2, 3, M'}. It is enough to show the 
result for i under the above assumption. 

First, writing i — 1 in place of i in (4.14), we have 

Pr{(Ft i ,ft 1 >^') i T^\Y h ^Z h ^V)} < eU (4.15) 

where e-_ x — > as e' — > 0. Further, noting that F« — > (y^j, V) — > Z/ i _ 1 forms Markov chain 
and using Lemma 4.4, we have, from (4.15), 

PrKPtX^') i T^\Y^Z h ^V)} < e'U (4.16) 
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where e"_ x — > as e' — > 0. Hence, of course, 

Prm'X'^V"') i T}r'\Y^Z h ^V)} < e'U- (4.17) 

Also note that condition (4.10) is same as condition (4.4) with (Y^, (Zj t _ 1 , V), Zj) in place of 
(Y"i, I2, Zi) ((^/j_i, V") — > — > Zj, of course, forms Markov chain). As a result, by Lemma 
4.2, there exists a sequence of mapping pairs (fi,gi) such that 

1 log |Z4| < i2j + ej (4.18) 

Th 

Pr{(>f , zf,§ ^, V»') £ 7>V*, ^ V)} < (4-19) 

* n' — n ' 

where Z i = gi(fi(Y 2 n '),(Z IiV V n ')) and e- — > 0. Further, noting the subtle fact that 
Y"j. _j — >■ (Y h Z I ._ 1 , V) — > Zj forms Markov chain and using Lemma 4.4, we have, from (4.16) 
and (4.19), 

Pr{(T;[X,V^^Ty\Y h ,Z h ,V)} < e'l (4.20) 

where e" — > 0. Hence (4.18) and (4.20) give (4.13) and (4.14), respectively, with max{e.,e"} 
now playing the role of e' { . 

Recall that we have / : y n >-^>U. Further, recalling 

Zt = g 2 (MY 2 n '),z"',V n ') 

= g2(f2(Y 2 n '),g 1 (f 1 (Y 1 n '),V n '),V n ') 

and so on, we have 

T =g(J(Y n '),V n ') 

, n > _ 

for certain g : U x V™ — > Z . Thus, for the mapping pair (f,g), (4.13) and (4.14) hold for 

each % G Im 1 - Writing e" = maxj{e-} and collecting condition (4.13) for each % G Im>, we get 

(4.2). Further, for i = M', (4.14) is (4.3). Hence r' G B. □ 

Now we are ready to complete the proof of Theorem 4.1 by showing B* C B. 

Proof of Theorem J^.l: By Lemma 4.5, any R' , such that R,'*^ < R'^ % G Im', 
belongs to B. Generalizing this by symmetry, any R ^\ such that R 1 *^ < R'}^, i G Im 1 , for 
arbitrary permutation ir G n, belongs to B. Further, by (4.9), any R G B* can be written 
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as a convex combination of at most M'\ such R' n 's. Since each such R,' n also belongs to 
£>, we have R,' G B due to convexity of B. Hence B* C B. □ 

Recall that we assumed statistical dependence of the random variables (Y, Z, V) in the 
above proof. However, Theorem 4.1 holds even when admissible subsets of (Y,Z,V) are 
independent. In such case, some of the constraints given in (4.1) degenerate. However, 
one can still identify the desired corner points of the resulting B* (which of course remains 
convex) and prove Theorem 4.1 mimicking our analysis. 



5 Proof of Theorem 3.1 for Problem 'TPC 

At this point let us turn to Theorem 3.1 (A = A*), which we shall prove for problem 'TPC. 
For the sake of convenience, we shall drop the subscript 'TPC throughout this section. 
The proof consists of two parts: The inner bound A D A* is shown in Sec. 5.2 using the 
fundamental principle given in Theorem 4.1. The outer bound A C A* is shown in Sec. 5.3 
with the aid of Slepian-Wolf theorem [4] and Fano's inequality [10]. For ease of reference, 
we first reproduce in Sec. 5.1 the definitions of A and A* for problem 'TPC from Sees. 2.4 
and 3.1, respectively. 



5.1 Definitions Reproduced 

Recall that (X,W,S) ~ p(x,w,s) and {(X(k),W(k), S(k))} are drawn i.i.d. ~ p(x,w,s). 
The subset Xj, J C 7 M , of sources are losslessly decoded and the complementary subset X JC 
is decoded under L bounded distortion criteria d\ : X 2 JC — > [0, di max ], I G II- Any (R, D) G A 
if for any e > there exists encoder mappings 

fm '■ Zmi m £ I M 

fM+ m ■ W™ -> Z M+m , me I K 

(i.e., / : ~X a x W" >— > Z) and decoder mapping g : Z x S n — > ~X n such that (reproducing 
(2.13), (2.9) and (2.10), respectively) 

-log|Z m | < R m + e, meI M +K (5.1) 
n 

PriTj^Tj)} < e (5.2) 

-EdmQTjcTjc) < A + e, I eh (5.3) 

Th 
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where X = g(/(r,lf), S n ). 

Further, any (R, D) G A* n if there exist product of (M + K) alphabets Z (with the re- 
striction Z m = X^, m G J), conditional distributions q m {z m \x T } n ) ) m G J c , and rj(zM+j\iVj'), 
j G Ik, and mapping ip : Z — > Afj c such that (reproducing (3.1)-(3.3), respectively) 

1 — n — — 

-/(X 7 ;Z 7 |Z WU ,^) < R T , ICI M \{} (5.4) 

i/(W;;Z M+ /|^M + / K \/^ n ) < Rm+i, IQIk\{} (5.5) 

-d ln (T Jc ^(Z,S n )) < D h l eh (5.6) 

where 

= r; (5.7) 

(reproducing (3.4)) and Pnix 11 , W 1 , s n ) = Y\^ =l p{x(k) ) w(k) ) s(k)). Moreover, we have seen 
that, splitting 7 = I' U I" C 7 M \ {} such that 7' C J and 7" C J c , we can equivalently write 
(5.4) as (reproducing (3.5)) 

-77(X / ,|X J \ / ,, ^/ M+K \(ju/")? £"0 + - Zi"\Xj, Z Im+k \( JuI »), S n ) < Rp + i?///. (5.9) 

Note that condition (5.7) affects neither (5.9) nor (5.5). Hence, from the set of conditions 
defining A* w we remove (5.7) and the requirement Zj = A 7 ^ by absorbing it in (5.6) so that 

-d ln (Tj^(Xj,ZiM + K\J,S n )) < D h I eh- (5.10) 

71/ 

Finally, recall A* = U™ =1 A* n . 



5.2 Inner Bound A. 3 A? 

For any (R,D) G A* = \J™ =1 A* n , (5.4)-(5.8) hold for some n, {q m }, {rj} and ip. Now, 
referring to Theorem 4.1, identify (Y,Z,V) with (W™ , Z m+i k , S n ) and 7? with nR M +i K , 
and note that condition (5.5) is same as (4.1). Consequently, by Theorem 4.1, for any 
e' — > 0, there exists a sequence of mapping pairs 

Um+i k '■ W ^ U M +i K ,g' ■ U M +i K x S nn -> 2 M +7k) 
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(for some n' — > oo) such that (4.2) and (4.3) hold. In other words, we respectively have (the 
first condition (4.2) is divided throughout by n) 

—. log \U m \ < R m + e'Jn, meM + I K (5.11) 

nn' 

Py{£} < ei (5.12) 



where 



{(TT n , z m+Ik , s nn ') i ^(W 1 , z M+lK , s n )} 



z M+ i K = g'ifM+i^ls™') 

and e[ — > 0. 

Referring to Theorem 4.1 again, now identify (Y,Z,V) with (X n , Z i M , (Z m+i k , S n )) 
and R with nRj M , and note that condition (5.4) is same as (4.1). Further, noting X™ — > 
(H^™, S*™) — > Zm+i k forms Markov chain and applying Lemma 4.4 in view of (5.12), we 
obtain 

Pr{pr n ', W^Xt+i^S™') i T^Or^^M^S")} < e' 2 
where e' 2 — > as e' — > 0. Hence, of course, 

nn' ^ ra ' / ' n 

Pr{(X ,Z M+lK ,S nn )^T e , (X ,Z M +i K ,S n )} <e' 2 . 

Consequently, by Theorem 4.1, for the same e' as earelier, there exists a sequence of mapping 
pairs 

(fi M ■■ " U lM ,g" : Z7 /M x (r M+lK x 5""') - Z?J 

(for some n' — > oo) such that (4.2) and (4.3) hold. Specifically, we respectively have (the 
first condition (4.2) is divided throughout by n) 

1 

nn' 



-log|Z4j| < R m + e' 3 /n, m e I M (5.13) 
Pr{8'} < e 3 (5.14) 



where 

(n')/Y^ 



£' - {(^ > ^/ M ' ( z m+i k > >S" m )) ^ Tj 1 [X , Z/ M , S n ))} 



and £3 — > 0. 
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Recall Z j = X 3 (as mentioned in (5.7)) and write Xj = Zj so that, from (5.14), we 
immediately have 

; ^ nn' 

Pr{X7 ^Xj }<e' 3 . (5.15) 

Further, denote 

X JC (j) = ij(Z(j),S n (j)), jel n , (5.16) 
Hence, from (5.14), we have 

/ ^ nn 1 , 

pr{^} = Pr{(r;: ,x JC ) i Tj n) (r; c ^(z,s n m < 4. (5.17) 

Now note 

— - f di(n n ')ffi?\T™ )<Di + e'^max, I e h (5.18) 

TXrfb 

for any (a?}™\x™ ) G 7j n ^(X^ c , ^(Z, S" 1 )). Hence, we obtain (for each Z G Jl) 

— Ed, (nn0 (X JC ,X JC ) < (1 - Pr{5i})(A + eUmax) + Pr{£iMmax 

< A + (e' + 4) rf miax (5.19) 

due to (5.17). 

^ n 

At this point, observe that, stacking X JC (j), j G in (5.16), we can write 

"Xj c =^'(T ,S nn ) (5.20) 
for certain mapping tp'. Hence, write 

^ nn' ^ n n' ^ nn' 

X = (X, ,X JC ) 

' — -7l' ' — ■ 7?/ ^™^7i' 

= (^'((^ M ,^M + ,J,S™')) 

and recall J C J M as well as 



-n 



= /(// M (* M^M + /^)) 

*W = g\f M+ i K (w^ n ),s nn ) 



to conclude that X = g(f{X nn , W nn ), S nn ') for certain g : U x S nn ' -> A'™" . Moreover, 
for any e > 0, choose e' > such that max{e' l5 e' 3 , (e' + 63)<i max } < e. Consequently, for 
mapping pair (/, g), conditions (5.11) and (5.13) give rise to (5.1), (5.15) to (5.2) and (5.19) 
to (5.3), with (U,nn') now playing the role of (Z,n). Hence (R,D) G A. In other words, 
A 2 A*. Since A is closed, we have A D A* (noting A* is the smallest closed set with A* 
as a subset). This completes the proof. □ 
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5.3 Outer Bound A C A* 

The proof requires Fano's inequality, a weakened version of which states the following: Given 
random variables U and V, 

H(U\V) <l + \og\U\Pr{U ^ g(V)} (5.21) 

for any g : V -> W [10]. 

Now consider any (i?, D) e A Then, for any e > 0, by definition, there exists mapping 
pair 

(f -.T 1 xW >-^Z,g :Z x S n -^T 1 ) 

of some length n such that (5.1)-(5.3) hold. We can further encode Z m+Ik = f M+i K (W n ) m 
a noncooperative manner with complete side information S n using interposed Slepian-Wolf 
code 

(fM+i K '■ Zm+i k ^ Um+i k ,92 '■ Um+i k x S nn — > Z m+Ik ). 
Given (R, D) £ A, e and (/, g), any rate- vector R' m+Ik is said to be achieved using interposed 
codes of the form {f' M +i K i 92) if f° r an y e' > there exists such code (of length n') that 
satisfies 

-\og\U m \ < R' m + e', meM + I K (5.22) 
n 

MT M+lK ^g'S' M+lK (Z n M+lK ),S nn ')} < e'. (5.23) 

In view of (5.1), setting / m+Ik to identity mapping (clearly, n! = 1, Um+i k = Z M +i K ) and 
choosing g2(f' M +i K (Z n M+i K ),S nn ') = 7 M+lK (Z r M +lK ) = ^m+Ik, o f course, (5.22) and (5.23) 
trivially hold for R m+Ik = ti{Rm+i k + e) irrespective of e'. Therefore, we have 

^(7m + /(^)I7a/W W L\/)> 5") < ^(/?m + , + 6) < + ATe, IQI K \ {} (5.24) 

by Slepian-Wolf theorem [10] . 

Similarly, we can also encode Zj M = / /m (X™) with complete side information (Zm+i k , S n ) 
(recall Z M +i K = f m+Ik^W™)) using interposed Slepian-Wolf code 

(f' lM : ~ U lM ,g[ : U Im x (T m+Ik x S nn ') - Z?J. 

Again, given (R,D) E A, e and (/,<?), any rate- vector i? /M is said to be achieved using 
interposed codes of the form (fi M ,g[) if for any e' > there exists such code (of length n') 
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that satisfies 



TV 

,-d 



■^log|W m | < R' m + e\ meI M (5.25) 



MK 1 ^9[(f I jZ n lM )>(Z r M +lK ,S nn '))} < e'. (5.26) 
In view of (5.1), setting fj to identity mapping (clearly, n' — 1, Uj M — Z Im ) and choosing 



9i ( fiM ( Z'i M ) ,{Z"m+i k > S nn ' ) ) = fijKt) = Z L, of course, (5.25) and (5.26) trivially hold 
for R Im = n(Ri M + e) irrespective of e'. Therefore, we have 

lHUAX)\liM' r iM\i)JM + i K (W n ), S n ) < + < Ri + Me, I C I M \{} (5.27) 

tei 

by Slepian-Wolf theorem [10]. 

At this point, let us write g = (gj, gjc) such that the ranges of gj and gjc are contained 
in Xj and X JC , respectively. Now, noting Xj = gj(f(X ,W ),S n )) and applying Fano's 
inequality (5.21) for U = X " and V = (/(T, W"), S n ), we have 

_ ^ n 

H(Tj\f(X, W), S n ) < 1 + nlog \Xj\ Pr{X n j + X j)} < n(l + log \Xj\)e. (5.28) 

In the above, the second inequality follows by (5.2) and by choosing n > 1/e. Next, refer to 
(5.27) and split each I = V U I" C I M \ {} such that J'CJ and I" C J c so that 

= ^(x;„7^(x;,) j /w ^)I7/ m \/ (^L^Jm+^cw* 1 ), s») 

- ^H^lJtjX^JJ^W 1 ), S n ) (5.29) 
due to the chain rule of entropy. From (5.28), note that 

H^lfjJJTjJJ^W 1 ), S n ) < n{\ + log \Xj\)e (5.30) 
(recall f(X,W) = (7, M (*L) Jiz+i^)))- Further, we can write 
/J(r;„ I7/ mU PC mU ), 7m+/ k (W 7 "), n 

= H(Xj,, f J"(X jrr)\f J\I'(X f I M \(JUI")(X I M \(JUI") )Jm + i k (W ),S n ) (5.31) 

> H(X r ,f pf (X III )\Xj^ II ,f lM ^j UIII) (X lM ^j UI ,^),f M+lK (W ),S n ) (5.32) 

— H(Xp\Xj^ Ih // m \(ju7»)(^7 m \(ju7"))' Im+i k ( w ),S n ) 

+ ^(7/»(A^;0l^,7^ A ( JU /'o(^M\(Ju/'o)'7M + / K (W 7n ), S n ). (5.33) 
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Here (5.31) follows by noting I M \ I = ( J \ 7') U (7 M \ ( J U 7")), (5.32) follows due to data 
processing inequality and (5.33) follows by the chain rule of entropy. Using (5.33) and (5.30) 
in (5.29) and rearranging, we obtain 

-H(X II \X J ^ I ,, // m \(ju/'')(^/ m \(ju/"))> fM+i K (W ), S n ) 

+ -H(fi"( x i")\ x ji fi M Wui")( X i M \(Jui"))i fM+i K ( W )>S n ) 

< R r + R ri + (M + l + \og\Xj\)e, I' C J, I" C J c , I' U I" C 7 M \ {}■ (5.34) 

Further, from (5.3), we have, for each / e 7 £ , 

= ^E^(X: c ,^(^,7 /M (X; M ),7M + / Jf (W 7n ),^)) (5.35) 

1 

> (1 - Vv{X n j ± XjV-EdUJT^g'AT}, frjX^JJ^W 1 ), S n )) 

> ^Ed lB (^ c ,^(^J /M (^) Jm^CW"),^)) -ed,max (5.36) 

> iEdtoCr^.^CYj, (7jffi)J /lf \Av))J^ (^)» ^)) - e ^ax 

= iEdfaCr^,^, (7 W (^y) Jw*^)), - «W (5.37) 

n 

where (5.35) clearly holds for suitable g' JC , (5.36) follows because Pr{Xj ^ Xj} < e (by 
(5.2)) and (5.37) clearly holds for suitable ip. Rearranging (5.37), we obtain 

-E^^X^ c ,^(X^,(7 /MV (X; MV ),7M + / x (W rn )),^ t )) < A + (l + ^max)e, I e I L . (5.38) 

At this point define for any e > and any integral n > 1 the set An ^ of rate-distortion 
pairs (R, D) such that there exist product Z of M + K alphabets (of which Zj can be 
arbitrarily chosen), conditional distributions qm(zm\%m), m £ J c \ an d rj(zM+j \ w j), J £ ^A", 
and mapping ip : ,Yj x Zj m+k \j A 7 ^ such that 

-H(X r \Xj\ r , 2 , / M+if \(j u /") ! 5"™) + -I( x i"'i z i"\Xj, Z Im+k \( Ju p>), S n ) 

< R P + R r > + (M + 1 + log \Xj\)e, 

I' C J, 7" C J c , 7' U 7" C I M \ {} (5.39) 

^7(Tu7;Z M+/ |Z M+/KU ,S n ) < R M+I + Ke, ICI K \{} (5.40) 

-rf ta (X^ c ,^(r;,Z wv ,5 n )) < A + (l + djmax)e, Z G I L (5.41) 
n 
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where 



(x n ,Tr,^,z /M+ifV ) ~ ^/.onu^w n^K)' (5.42) 

Comparing (5.39)-(5.42) with (5.9), (5.5), (5.10) and (5.8), respectively, note that A*^ = 
A* n . Also note that X (ei) C for e 1 < e 2 . Further, let .A*( £ > = U^iX W . Of course, 

^4*(ei) c ^.*( £2 ) for ei < e 2 . Hence, noting f > | e>0 ^4.*( e ) is closed, we obtain 

p|^y = ^W=^. (5.43) 

The second equality in (5.43) holds because A* {0) = lX=i^ (0) = LC=i A l = 

Finally, consider any (R, D) G A. Recall that for any e > there exists mapping pair 
such that (5.34), (5.24), and (5.38) hold. Choosing 

and keeping the present ip, note that the above three conditions coincide with (5.39)-(5.41), 
respectively. Also, (5.42) holds for g m (z m |x™ ) = 5(z m — re™ ), m G J c , and rj(z M +j\w 7 -) = 
S(zM+j — w1 j), 3 ^ Ik, where 5 (a — b) = 1 if a = b and 5(a — b) = otherwise. Hence 
(R,D) G An (t) - Consequently, we have (R,D) G A* (e) C A*^ for each e > 0. Hence, by 
(5.43), we have (R,D) G f] e>0 = A*. This completes the proof. □ 



6 Application to Estimation Theory 

Next we pose the entropy-constrained estimation problem in a multiterminal setting. We 
shall show that our theory of source coding solves this problem as a special case. 

Problem Statement: Consider estimation of X on the basis of observations W = 
(Wi, W 2 , W K ) available at base station at respective rates R = (i?i, i? 2 , Rk)- In ad- 
dition, let observation S be completely available at the base station. The estimation error 

2 

is measured using a bounded distortion criterion d : X — > [0,d max ]. Formally, suppose 
(X,W,S) ~ p(x,w,s) and draw {(X(k), W(k), S(k))} i.i.d. ~ p(x,w,s). We encode W 
using K encoder mappings 

f m :Wl^Z m , me I K (6.1) 
(i.e., / : W™ >— > Z) and decode using decoder mapping 

g :Z x S n X n . (6.2) 
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In other words, estimate X n by 

X n = 0(700,5") (6.3) 

with corresponding estimation error ^Ed n (X n , l"). The achievable set ^4 E is defined by the 
set of pairs (i?', D) such that, for any e > 0, 

-log|Z m | < R' m + e, meI K (6.4) 

77/ 

-Ed n (X",X n ) < D + e. (6.5) 
n 

Next we give an information-theoretic description of A E . 

Achievable Region: First of all, refer to Sec. 2.4 and note that the estimation problem 
at hand is a special case of the source coding problem 'DPC. Specifically, the number of 
sources X is M = 1 and the number of distortion criterion is L = 1. Further, X is not 
encoded, i.e., rate Ri = 0, and R now plays the role of Ri+i K - Hence, by Theorem 3.1, we 
have 

A E = <A^i 

where A* E = U™ =1 A* nE , 

A* nE = {(Ri + i K ,D) : (R, D) e A* nDPC , R l = 0} (6.6) 

and A* mPC is defined for M — 1 and L — 1. Further, refer to the conditions (3.13)-(3.16) 
defining A* nT)PC and use R\ = 0. Note that condition (3.13) and the auxiliary random 
variable Z 1 are redundant. Hence, Z x can be marginalized out from (3.16). For an explicit 
definition of A* lEj equivalent to (6.6), let us rechristen Zi +Ik as z' and rewrite (3.14)-(3.16) 
in the modified form. In particular, a rate-distortion pair (R\ D) belongs to A* m if there 
exist product of K alphabets z' , conditional distributions rj(z'j\w^), j G Ik, and mapping 
ip : Z' — > X n such that 

-IiWl-z'^S") < R' I} ICI K \{} (6.7) 

n 

-d n (X n ,ij(z',S n )) < D (6.8) 

where 

(X n ,W n ,S n X)~Pn(x n ,w n ,s n ) J] r^z'lw]) (6.9) 

jei K 

and p n (x n ,w n , s n ) = Ylk=i p(x(k),w(k), s(k)). This solves the entropy-constraint estimation 
problem. 
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7 Conclusion 



In this paper, we presented a unified solution to all multiterminal source coding problems 
where encoders do not cooperate, encoded information is jointly decoded and the distortion 
criteria, if any, apply to single letters. In particular, we unify all admissible source coding 
problems, irrespective of number of sources and availability of side information, using a 
fundamental principle (Theorem 4.1) based on typicality. The power of the above principle 
comes from a novel dissociation of distortion criteria from the core of source coding problems. 
In a way, our work marks the culmination of decades of source coding research pioneered by 
Shannon [2, 3] and enriched by works of Slepian-Wolf [4], Wyner [5], Ahlswede-Korner [6], 
Wyner-Ziv [7], Berger et al. [8] and Berger-Yeung [9]. At the same time, our result clears 
the path for new research which hitherto seemed too difficult to attempt. The multidecoder 
extension of our theory is of course the natural next step. Another open problem that 
also comes to mind is characterization of the achievable region for the entropy-constrained 
detection problem in the multiterminal setting. The main difficulty in this problem is that 
natural performance measures of detection, such as Bayesian probability of error, are not of 
single-letter type. We also believe that the ongoing research into channel coding theory will 
receive certain direct and indirect clues from our work. In the least, researchers investigating 
the capacity regions of not-so-well-understood channels, such as the broadcast channel, will 
now be open to the possibility of a higher order information-theoretic description instead of 
the usual first order. 



27 



A Equivalence of (3.1) and (3.5) 



Recall Zj = Xj and I = V U I" C I M \ {} such that KJ and 7" C J c . Hence, we can 
write 

I{ x i\ z i\ z i m +k\Ii S n ) = I(Xr,X I „;X I ,,Z I »\Xj\ I ,,Z lM+K \(j LII »- ) ,S n ) (A.l) 

= /(^;^,,Z 7 »|T; v ,,C/)+7(^;^,Z / »|T;,C/) (A.2) 
= (X?,|T; U „ C/) + 7(X7„; Z^K, C/) (A.3) 

where (A.l) follows by noting I M+K \ I = (J \ 7') U (7 M+isr \(JU 7")), (A.2) follows by the 
chain rule of mutual information and by denoting U = (Z Im+k \( JuI h}, S n ) and (A.3) follows 
because V C J. In view of (A.3), (3.1) and (3.5) are indeed equivalent. 

B Proof of Lemma 4.3 

B.l Information- Theoretic Relations 

First we need certain information-theoretic relations involving 

M' 

(Y, Z,V) ~p'(y,v) Y[<l'm( 

m=l 

where all random variables are dependent. 

Lemma B.l Suppose sets 1,1' C I M , \ {} are disjoint. Then 

I (Yj; ZjlZ^y, V) = I {Yj; Z^Zjc, V) + 7 (Z 7 ; Z r \Z {Iuir , V) . (B.l) 

Proof. First expand 

7 (Z /; F 7 , Z r \Z {Iuir , V)=I (Zr;Z r \Z {Iuir , V) + 7 (Z^F^Zjc, V) , (B.2) 

applying the chain rule of mutual entropy. Expand the same quantity again, now applying 
the chain rule in a different order: 

7 (ZjjFj, ZV|Z" (JU/ ,)c, y) = 7 (ZjjFjIZ^c, + 7 (Z 7 ; Z r |F 7 , Z (JUJO c, V) . (B.3) 

Note that Z/ — > (Yj, Z V) — > Z// forms Markov chain, i.e., 7 (Zj; Z//|Yj, Z^py, V) = 
in (B.3). Hence, equating right hand sides of (B.2) and (B.3) and rearranging, we obtain 
(B.l). □ 
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Lemma B.2 Suppose sets 1,1' C 1 M , \ {} are disjoint. Then 

l(Y Iur ;Z Iljr \Z {Iuir ,V) = l(Y I ;Z I \Z iIuir ,V)+l(Y I r,Z I ,\Z r c,V). (B.4) 

Proof. For any quadruple (U 1: U 2 ; V 1: V 2 ) of random variables, we can write 

IiU^U^V^Vt) = I(U 1 ,U 2 ;V 1 ) + I(U 1 ,U 2 ;V 2 \V 1 ) 

= I(U 1 ;V 1 )+I(U 2 ;V 1 \U 1 )+I(U 2 ;V 2 \V 1 )+I(U 1 ;V 2 \V 1 ,U 2 ) (B.5) 

by repeatedly applying the chain rule of mutual information. Using formula (B.5), we obtain 

I (Yiui r , Z iui'\Z \iui') c , V) = I {Yj; Zj\Z( IUI <y, V) + I (Y P ; Zj\Yj, Z( IuP y, V) 

+ I (F r ; Z r \Z r c, V)+I (Yj; Z P \Y P ,Z P c, V) . (B.6) 

Here I (Y P ; Zj\Yj, Z^ IvjP y, V) = and I (Y p Z P \Y P , Z P c, V) = 0, respectively, because 
Zi — > (Yj, Z( IyjP y, V) — > Yj/ and Z P — > {Y P , Z P c,V) — > Yj form Markov chains. Hence 
the result. □ 

More generally, any I C I M , \ {} can play the role of Im 1 in the statement of Lemma B.2 
so that I' c can be replaced by I \ I' and (I U I') c by 7 \ (7 U 7'). In that case, Lemma B.2 
immediately takes the form: 

Corollary B.3 Suppose set 7 C I M , \ {} is given and sets 1,1' C I are disjoint. Then 

i{y iup -Zj up \z Hiuiiv v) = /(F J ;Z J |z A(JUJ , ) ,y)+/(y r; z r |z Ar ,y).(B.7) 

Now consider any 7 C I M , \ {} with cardinality \I\—m and write 7 = {7(1; m)}. Further, 
setting I = I M ' and letting (-{7(1)}, 7 \ -{7(1)}) play the role of (7, 7') in (B.7), we have 

I {Y I ;Z I \Z I c,V) = / (F^i); Z i( i)|Z/ c , V) + / 1/) . (B.8) 

Noting 7 \ (i(l)} = {i{2 : m)} and continuing the recursion by letting ({i(2)} , I \ {7(1; 2)}) 
play the role of (7, 7') in (B.7) and so on, we obtain 

m 

l{Yj;Zj\Z P ,V) = ^7(y (j) ;Z, (j) |Z (A{i(1:j _ 1)}) c,y). (B.9) 

i=i 

Noting (7 \ {i(l : j — 1)}) C = Im' \ '■ m)} in (B.9), we have the following: 
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Corollary B.4 For any set I = {i(l; m)} C I M , \ {}, 

m 

I^j-Zj\Z IC ,V) = J2H Y ^ Z i(J)\Zl Ml \{i(j:rn)},V). (B.10) 

Further, suppose I — I m for some 2 < m < M' . For the choice / = J m _i and i 7 = {m}, 
(B.7) becomes 

/ (F Jm ; Z Im \V) = I (F /m _ l5 Z /m _JV) + / (y m ; Z^Z^, V) , (B.ll) 
which gives a useful chain rule. Applying this repeatedly, we obtain: 

Corollary B.5 For any 2 < m < M' , 

m 

l{Y Im ;Z Im \V) =Y,Ip*Zi\Zi^V). (B.12) 

i=i 

In fact, corollary B.5 can be further generalized as follows. For any 1 < m < M', set 
/ = Im'i I = I m and I' = Im> \ I m in Lemma B.3 to obtain 

1 { Y r M r,Z lMl \V) = I {Y Im ;Z Im \V) + / {Y hAIm] Z hAIm \Z Im , V) . (B.13) 

Expanding I {Yj Zj ,\V) and I (Y Im ; Z Im \V) using Corollary B.5, from (B.13) we obtain: 

Corollary B.6 For any 1 < m < M' , 



M 



I {Y lMlVm ; Z lMl \ Im \Z Im ,V) = £ / (Yf, Z^Z^V) . (B.14) 

i=m+\ 

We require one more information-theoretic relation. 

Lemma B.7 For any I C I M , \ {}, 

I (Yj-Z^Z^V) <J2 r faZilZh-^V) . (B.15) 
iei 

Proof. Let m = \I\ and write the elements of / = {i(l), i(2), i(m)} in ascending 
order. Hence, we have 

h(j)-i C I m> \ : m)}, j E I m . (B.16) 
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For each j, denote 

= ( j m> \ ■ m)}) \ 7 i(j) _!. 

Hence we can write 

1 ( Y i(fh Z Hj)\ Z I M '\{i(j™)}, V) = I (V; (i) ; Z i(j) | , \/) 

= H {z i{j) \Z IiU) _^ Z }(j) , V^j - H (z i{j) \Y i{j) , Z h{]) _^Z~ I{ . y V^j 
= H [Z^Z^Z^ V)-H (z iU) \Y iUh Z Ii(j) _ i: v)(B.17) 
< H (Z^Z^V) - H {Z^Y^Z^V) (B.18) 
= l{^UhZ i{j )\Z ItU) _ 1 ,v). (B.19) 

Here (B.17) follows by noting 

H ( z i(j)\ Y iU)i z h-i, z i(j), V) = H (Z i{j) \Y i{j) ) = H (Z i{ j)\Y i{j) , Z h _ x , V) 

due to the fact that Z^ — > Y^ — > (Z Z j^,V) form Markov chain. Further, (B.18) 
follows because conditioning reduces entropy. Now summing (B.19) over j e I m , we obtain 

m m 

1 ( Y i(jh z i(j)\ z i M >\{i(j--™)}' v ) <z^! ^(j)l^o)_i^) • ( B - 20 ) 

3=1 3=1 

By Corollary B.4, the left hand side of (B.20) equals / (F 7 ; Z/|Z/c, V^). Also, note that the 
right hand side of (B.20) is same as the right hand side of (B.15). Hence (B.20) is the desired 
result. □ 

Now let us turn our attention to finding the corner points of B* . 



B.2 Properties of Active Constraints 

Lemma B.8 Suppose 

ifY^ZjlZ^V) = R'j (B.21) 

I (Y r ;Z r \Z r c,V) = R'j, (B.22) 

simultaneously hold for distinct sets 1,1' C Im\{}- Then either I C I' or I' C /. 

Proof. It is enough to show that if I \ I' ^ {} as well as V \ I ^ {} then there exists 
no rate vector r' G B* such that (B.21) and (B.22) hold simultaneously. To prove this, first 
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we assume that (B.21) and (B.22) hold for some R,' E B* and some (I, /') with the above 
property and then detect a contradiction. 

Two cases arise depending on whether I and I' are disjoint or not. First consider the 
case where I (1 1' = {}. Using (B.l) in (B.4), we obtain 

l{Y Iur ;Z IuP \Z {Iuir ,V) = 7(F / ;Z / |Z J c,y)+7(F J ,;^|^c,y) 

+l(Zj;Z r \Z {Iuir ,V). (B.23) 

Now, adding (B.21) and (B.22) and comparing with (B.23), we have 

R'lur = I {Yiup'i Ziui>\Z( IVJI iy, V) — I [Z p, Z p\Z ( Iur y , V) 

< I (Y IUI ,;Z IUI ,\Z {Iuir , V) (B.24) 

because (Zj, (Z( lLII /y, V), Zp) does not form Markov chain. Note that (B.24) contradicts 
(4.1) where of course I Li I' now plays the role of /. 

Next consider the case where I f] I' — I ^ {}. Writing I — (I \ I) U /, from (B.21), we 
have 

R'^ + R'i = I (Yf,~Zi\Zic, V) 

= I (F A/ ; Z^\Z (iW , v) + I (F /; Zj\Z Iei V) + / (Z A/ ; v) (B.25) 

which is same as (B.23) with (/ \ /, I) in place of (I, I'). Further, from (4.1), note that 

R'^I^j-ZjIZj^V). (B.26) 

Using (B.26) in (B.25), we have 

R'^i < I (F A/ ; Z^\Z m c, V) + / (Z A/ ; Zj\Z^ v) . (B.27) 

Adding (B.27) and (B.22) and comparing with (B.23) (with I \ I now in place of J), we 
obtain 

R'lur < I (Yiui r , % i\jv\Z {i\jvy ■> V) 

-/ {z ixi -Z P \Z mr ,v) +1 (z^ZjIZ^v) . (B.28) 

Further, expand 

^ Zr\Z( Iu j/y, V^j = H (^Zj^\Z^ Iur y,V^ — H (^Z I ^\Z I t u ^ Iu py,V^ (B.29) 

/ (Z A/ ; Zj\Z IC , V) = H (z ixi \Z IC , v) - H (z^\Z iyjIC , v) , (B.30) 
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note V U (I U I'f = IUI C and subtract (B.30) from (B.29) to obtain 

I {^ z i\d z i'\ z {iui') c , V^j - I (z^f ZjIZjc, V^j — H (z ^Z^py, V^j - H (^Zj^Zp, V^j 

= 1 ( K z i\f z i'\i\ z (i'Ji') c ' ^) (B.31) 
> 0. (B.32) 

Here (B.31) follows by noting I c = (I U I' f U (/' \ I) and (B.32) follows due to the fact that 
{Zj\j, (Z( Iu py, V), Zp^j) does not form Markov chain. Using (B.32) in (B.28), we obtain 
(B.24) again which contradicts (4.1) as earlier. Hence the result. □ 

B.3 Identifying Corner Points 

Lemma B.9 B* has at most M'\ comer points. 

Proof. Due to Lemma B.8, the number of corner points of B* is upper bounded by the 
number of possible ways we can have 

/« c J (2) c ... c J (m) c J (m+1) c ... c J (M '- 1} C I M >, 

|j(m)| = m; l < m < M' . Note that, given l( m+1 \ we can choose 1^ inm+1 possible ways. 
Consequently, we can choose {/ (m) }™= x in 2 x 3 x ... x M' = M'\ possible ways. Hence the 
result. □ 

Lemma B.10 The rate vector r' such that 

R\ = I(Yi- ZilZ^, V), i G I M > (B.33) 

gives a corner point of B* . 

Proof. From (B.33), we can write 

M' M' 

W;Z l \Z h _ 1 ,V)= % (B.34) 

i=m+l i=m+l 

for each m — 0, 1, M' — 1. Further, by Corollary B.6, (B.34) is same as 

J(F 7 , u :Z 7 , u |Zj ,V) = R' r u (B.35) 
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which makes M' constraints, given in (4.1), active. To see this, set / = I M > \ I m m (4-1), 
vary m — 0, 1, M' — 1 and compare with (B.35). Therefore, in order to establish R r as a 
corner point of B*, we are just left to show r! G B*. Towards that, note, by Lemma B.7 and 
(B.33), that 

I (Y r ,Zj\Z IC ,V) <J]/(y i ;Z i |Z J ._ 1 ,y) =B! I (B.36) 
iei 

for all I C I M , \ {}. Comparing (B.36) with (4.1), we conclude r' G B*. This completes the 
proof. □ 

Proof of Lemma 4-3: Now, noting symmetry, the indices {1,2, ...,M'} in (B.33) can 
be permuted to obtain M'\ corner points. Finally, by Lemma B.9, such enumeration of 
corner points of B* is exhaustive. Hence the result. □ 
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